Repository navigation
Conversation
… normalization normalizeRemoteSession() dropped closedTabTombstonesByTabId, so both pull entry points (notification and get paths) never saw remote tombstones and remoteWorkspaceSessionMatchesSnapshot could not detect a tombstone-only delta. Also replaces the inline tombstone record type in WorkspaceSessionState with the shared ClosedTerminalTabTombstonesByTabId alias (stablyai#10342).
…p comments Clarify that a live tab beats its tombstone only for the current merge pass (hydration re-unions the store's tombstone map afterwards), explain why the final tabsByWorktree sweep is safe across the whole map, note the stop-to-poll behavior change in the flush reschedule comment, and add a test pinning that a tombstone wins over preserveLocalTerminalTabIds for a tab that is not live locally (stablyai#10342).
…on-field registries
33ebb6f to
0718fa6
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
🚧 Files skipped from review as they are similar to previous changes (2)
Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review. 📝 WalkthroughWalkthroughThe change adds closed terminal-tab tombstones to terminal state and workspace session persistence. Tombstones record deliberate closes, expire after 30 days, and are capped at 500 entries. Local and remote sessions now merge, normalize, import, and export tombstones. Direct-SSH merging filters tombstoned tabs, layouts, and mappings. Workspace hydration and profile field metadata include the new state. Session persistence now retains pending changes while scheduling is gated and flushes them when scheduling reopens. Merge Risk: ⚪ Minimal · up to The change records closed SSH terminal tabs so they remain closed across reconnects and ensures those close records are persisted; no actionable merge-blocking risk remains beyond normal checks and review. 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
Full details: Description checkExplanation The description is detailed and covers the change, rationale, linked issue, testing, AI disclosure, compatibility, and checklist. However, the required Visual Proof section lacks an AFTER recording or screenshot for this behavior change.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1b802a7e-bc70-40fd-904e-4e870246e027
📒 Files selected for processing (29)
src/main/ipc/remote-workspace-snapshot-normalization.test.tssrc/main/ipc/remote-workspace-snapshot-normalization.tssrc/main/orca-profiles/profile-project-session-field-disposition.tssrc/renderer/src/hooks/remote-workspace-session-merge-tombstones.test.tssrc/renderer/src/hooks/remote-workspace-session-merge.tssrc/renderer/src/lib/session-write-subscriber.test.tssrc/renderer/src/lib/session-write-subscriber.tssrc/renderer/src/lib/workspace-session-browser-history.test.tssrc/renderer/src/lib/workspace-session-closed-tab-tombstones.tssrc/renderer/src/lib/workspace-session-editor-drafts.test.tssrc/renderer/src/lib/workspace-session-host-field-ownership.tssrc/renderer/src/lib/workspace-session-liveness.test.tssrc/renderer/src/lib/workspace-session-patch.tssrc/renderer/src/lib/workspace-session-relevant-fields.test.tssrc/renderer/src/lib/workspace-session-tabs-sanitize.tssrc/renderer/src/lib/workspace-session-tombstone-roundtrip.test.tssrc/renderer/src/lib/workspace-session.tssrc/renderer/src/store/slices/terminals.tssrc/renderer/src/store/terminals/terminal-state.tssrc/renderer/src/store/terminals/terminal-tab-close-tombstone.test.tssrc/renderer/src/store/terminals/terminal-tab-close.tssrc/renderer/src/store/terminals/workspace-terminal-hydration-patch.tssrc/renderer/src/store/terminals/workspace-terminal-hydration.tssrc/shared/closed-terminal-tab-tombstones.test.tssrc/shared/closed-terminal-tab-tombstones.tssrc/shared/remote-workspace-session-projection-tombstones.test.tssrc/shared/remote-workspace-session-projection.tssrc/shared/remote-workspace-types.tssrc/shared/workspace-session-state-types.ts
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.
…ctive pointers to tombstoned tabs Reject malformed closedTabTombstonesByTabId entries (null, missing worktreePath, non-numeric closedAt) during inbound snapshot normalization instead of trusting the shape all the way to importRemoteWorkspaceSession, and null out activeTabId / activeTabIdByWorktree entries that name a tombstoned tab as defense-in-depth. Also add docstrings to the exported tombstone helpers this branch introduced.
An enterprise user: "Every day I open orca and it opens more tabs daily at a linear scale." Three reports over a week, told on 08-19 that a PR had fixed it, reported twice more after. STA-4658 (P0), GH #12447, #15136, #10342, #9585. One install held 39 zombie tab records. The revived tab's sleeping-agent record still holds the pre-close session id, so it boots `claude --resume <old id>` -- two agents on one transcript. ## The chain, measured Reproduced deterministically in `ssh-lost-kill-tab-resurrection.spec.ts`: close an SSH tab, kill the relay daemon in the container so `pty.kill` rejects with a transport-class error, reconnect. drop 2 resurrected the closed tab <id>: baseline=1 drop1=1 drop2=2 (closed tab returned) drop3=1 The trigger is narrow and had to be measured rather than assumed: killed relay daemon reproduces **6 of 6 runs**; an orderly `ssh.disconnect` **passes**. Only an ungraceful loss -- network partition, host reboot, relay crash, a laptop sleeping mid-session -- strands the close with the RPC rejecting on a transport-class error. Both variants live in the spec behind one `runResurrectionCycles` parameterized solely by the disruption, so the difference is attributable to that single variable. What actually carries the tab back, from the pull path (`workspace.get` -> `getRemoteSnapshot`, `remote-workspace-relay-sync.ts:29`): pullSnapshot rev=3 tabs={repo:["16c4a3e1","06aba6b6"]} pullSnapshot rev=4 tabs={repo:["16c4a3e1","ff72768e"]} <- ff72768e IS the resurrected tab pullSnapshot rev=5 tabs={repo:["16c4a3e1","ff72768e","da21b76c"]} The client uploaded the session containing the tab; the user closed it; the kill RPC rejected so the close never reached the host; the host's snapshot still lists it; the client pulls it back and the merge restores it -- **correctly, by its own rule that the host is authoritative for what it knows.** A pane then mounts, respawns, and takes the recycled pty id. Client-side correlation from the same run, two controls and one positive in one run differing in exactly one variable: | Tab | Close events observed | Resurrected? | |---|---|---| | `6305cc07` | `user` + `pty-exit` | No | | `ed56f66c` | `user` + `pty-exit` | No | | `2036e760` | `user` only | **YES** | ## The fix `src/shared/closed-terminal-tab-tombstones.ts` (99 lines). A client-recorded close is first-party intent and must survive until the host acknowledges it. Per `docs/reference/ssh-execution-boundary.md` the remote verdict is `unverifiable` -- which may not authorise declaring the process dead, but equally must not authorise resurrecting the tab. This is SSH-v3 principle P2, "durable tombstones with a monotonic per-scope revision", reusing the existing `RemoteWorkspaceSnapshot.revision` rather than adding a twelfth per-tab identity field (the codebase carries eleven, 784 refs, that SSH-v3 Phase 3 deletes). - **Recorded** only on `closeReason === 'user'` (`terminal-tab-close.ts:69`). - **Suppresses** a host-sourced tab only when `tabId in tombstones && !currentTabsById.has(tabId)` -- a live local tab always wins, because deleting a live pane is the one outcome the merge exists to avoid. - **Retires** on positive acknowledgement: `!hostKnownTabIds.has(tabId) && hostRevision > observed`. Strictly newer, so a pull already in flight at close time cannot ack a close it predates. - Three never-retire guards: no revision retires nothing; a worktree the snapshot has no row for retires nothing; the first omitting snapshot only stamps the watermark. - TTL (30d) + cap (500) are **backstops** for a target the user never returns to, not the mechanism. - **Client-local only** -- never crosses the wire, so there is no mixed-version exposure. - Suppression is scoped to `replaceWorktreeIds`, which is what makes the live-tab check meaningful. A final whole-map sweep over the assembled `tabsByWorktree` would break that (a live tab is absent from `currentTabsById` outside the scope and would look suppressible); it is deliberately not there, and the comment at the top of the function says so. ## Evidence The load-bearing evidence is an A/B control on one tree, not the oracle's assertion. Flipping `isSuppressedByClose` to `false` -- one character -- reproduces the resurrection on demand: --repeat-each=2: 1) drop 2 resurrected the closed tab ab0e305d-…: baseline=1 drop1=1 drop2=2 2) drop 2 resurrected the closed tab 51533e34-…: baseline=1 drop1=1 drop2=2 2 failed With suppression on: **0 occurrences of "resurrected the closed tab" across five runs plus one independent run by a second agent.** Provenance verified positively, not by mtime: `closedTerminalTabTombstonesByTabId` appears 13x across 3 renderer chunks including `store-Do3KBvRE.js`; for every red control run `mayCreate` appeared 0 times in `out/main/index.js`. At the unit layer, disabling the same predicate: 3 failed | 39 passed. Restored: 42 passed; 287 across the workspace-session, terminal-store, remote-workspace, shared-tombstone and profile suites; 24 in the four tombstone suites. ## The oracle spec: GREEN in the full lane `ssh-lost-kill-tab-resurrection.spec.ts` passes both tests at this commit. Full Docker-SSH lane, clean tree: BUILD_SHA=49bb96e0b4c DIRTY=0 PROVENANCE tombstone=13 hasLocalTabsRow=2 hostAuthority=4 mayCreate=3 14 specs / 20 tests -> 17 passed, 2 failed, 1 skipped (10.7m) [12/20] :178 does not resurrect tabs whose kill was lost to a killed relay daemon PASSED [13/20] :190 does not resurrect tabs closed while the host is disconnected PASSED grep -c "resurrected the closed tab" (whole lane) -> 0 It passes WITHOUT PR 7 in the build (`mayCreate` present, `SshPtyAbsentFromRelayError` absent), so the bug-2 fix below is not required for it. Test 1 fails intermittently in ISOLATED single-spec runs, where a third defect blocks its cycle-2 setup. The resurrection assertion itself has never failed with this fix in place -- the intermittent failure is always a setup failure, never a resurrected tab. A reviewer running the spec alone may see it red; that is not this fix regressing. Three defects sit under STA-3374 and should not be conflated: - Bug 1 -- the closed tab resurrects. Fixed here. - Bug 2 -- `ssh-pty-session-reattach.ts:227-231` rewrites the relay's `PTY "pty-1" not found` into a bare `SSH_SESSION_EXPIRED`, so `isPtyAlreadyGoneError`'s `/PTY ".+" not found/` cannot match and `attachStablePaneOwner:242`'s already-correct fallback never runs. Owned by PR 7 (`nwparker/ssh-07-absent-from-relay`). Not required for the oracle above. - Bug 3 -- after the daemon is killed and the client launches a replacement, the client's OWN SSH transport drops and does not reconnect within 60s: no "delay step 2/9", no handshake failure, nothing. `ssh-connection.ts:1533` only logs on an SSH-level close. Unfixed, its own ticket. This is what makes test 1 intermittent in isolation. Discriminator for bug 3, measured in the isolated runs (the lane above ran without `ORCA_E2E_FORWARD_APP_LOGS=1`, so it was not re-confirmed there): `[ssh-relay] Socket probe result:` reads "DEAD" on every cycle of test 1 (daemon killed, a NEW relay must be launched) and "ALIVE" on every cycle of test 2 (daemon survived). Whenever a new daemon must be launched, the SSH transport drops afterwards and does not recover. An earlier reading blamed `kill.ts:82-84` for skipping `finishPtyShutdown` on a non-already-gone error. That was eliminated by direct test: the implied fix, `markSshRemotePtyLease(…, 'expired')` in that branch, was implemented, changed nothing, and was reverted rather than shipped unproven. Recorded so the path is not re-walked. The `SSH_SESSION_EXPIRED` rejection is real but fires during cycle 1 for the baseline pane, after which cycle 1 completes; the 60s silence begins only after `Relay channel lost ..., triggering reconnect`. The spec is claimed by the Docker-SSH lane, and that lane does not gate merges today. ## Divergences from open PR #16571 #16571 implements the same concept. Three deliberate changes: 1. It never retires on acknowledgement -- TTL+cap only, so it never converges. Ack retirement added. 2. It crosses the wire and lets a HOST-sourced tombstone delete a LOCAL tab in a final whole-map sweep. After #14361 that is the wrong risk; dropped. This also removes the mixed-version regression its own body flags. 3. Its hydration unions rather than replaces the map -- a union resurrects every tombstone the merge just retired, so it never converges. Its `activeTabId` nulling is also dropped as redundant: `workspace-terminal-hydration.ts:99-105,126-138` already revalidates both pointers against the tab rows it just built, and nulling twice would add a second rule that has to stay in step with the first. ## Can a tab the user did NOT close disappear? No, but the guarantee needs stating precisely. The only writer is `recordClosedTerminalTabTombstone` (`terminal-tab-close.ts:69`), reachable only on `closeReason === 'user'`; suppression additionally requires the tab not be live locally. Reopen (`recently-closed-tabs.ts:122-166`) calls `createTab` and restores cwd/shell/title/color/position, never the old id. **Caveat, stated because the slogan is not literally true:** `createTab` honours a caller-supplied id hint (`terminal-tab-creation.ts:53-65`, used by `useIpcEvents` for host-admitted tabs), so "tab ids are uuids that never recur" does not hold in this codebase. The guarantee rests on the `closeReason === 'user'` writer plus the live-local-tab check, not on id uniqueness. ## Risk Renderer-side, client-local, no wire change. The blast radius is `mergeDirectSshRemoteWorkspaceSession` and the persisted session field. Worst case if the ack logic were wrong in the retiring direction: a tombstone outlives its usefulness and suppresses a host tab whose id the host re-issues -- bounded by the live-local-tab check, the 30d TTL and the 500 cap. Worst case in the other direction is today's behaviour. `profile-project-session-field-disposition.ts` records the new field as `notRepoScoped` / `notTransferred` residue, bounded by the same TTL and cap. ## Verify pnpm test src/shared/closed-terminal-tab-tombstones.test.ts \ src/renderer/src/lib/workspace-session-closed-tab-tombstones.test.ts \ src/renderer/src/store/terminals/terminal-tab-close-tombstone.test.ts \ src/renderer/src/hooks/remote-workspace-session-merge-close-tombstones.test.ts To reproduce the bug this fixes, set `isSuppressedByClose` to `() => false` in `remote-workspace-session-merge.ts` and run `pnpm test:e2e:ssh-docker -- tests/e2e/ssh-lost-kill-tab-resurrection.spec.ts --repeat-each=2`.
An enterprise user: "Every day I open orca and it opens more tabs daily at a linear scale." Three reports over a week, told on 08-19 that a PR had fixed it, reported twice more after. STA-4658 (P0), GH #12447, #15136, #10342, #9585. One install held 39 zombie tab records. The revived tab's sleeping-agent record still holds the pre-close session id, so it boots `claude --resume <old id>` -- two agents on one transcript. ## The chain, measured Reproduced deterministically in `ssh-lost-kill-tab-resurrection.spec.ts`: close an SSH tab, kill the relay daemon in the container so `pty.kill` rejects with a transport-class error, reconnect. drop 2 resurrected the closed tab <id>: baseline=1 drop1=1 drop2=2 (closed tab returned) drop3=1 The trigger is narrow and had to be measured rather than assumed: killed relay daemon reproduces **6 of 6 runs**; an orderly `ssh.disconnect` **passes**. Only an ungraceful loss -- network partition, host reboot, relay crash, a laptop sleeping mid-session -- strands the close with the RPC rejecting on a transport-class error. Both variants live in the spec behind one `runResurrectionCycles` parameterized solely by the disruption, so the difference is attributable to that single variable. What actually carries the tab back, from the pull path (`workspace.get` -> `getRemoteSnapshot`, `remote-workspace-relay-sync.ts:29`): pullSnapshot rev=3 tabs={repo:["16c4a3e1","06aba6b6"]} pullSnapshot rev=4 tabs={repo:["16c4a3e1","ff72768e"]} <- ff72768e IS the resurrected tab pullSnapshot rev=5 tabs={repo:["16c4a3e1","ff72768e","da21b76c"]} The client uploaded the session containing the tab; the user closed it; the kill RPC rejected so the close never reached the host; the host's snapshot still lists it; the client pulls it back and the merge restores it -- **correctly, by its own rule that the host is authoritative for what it knows.** A pane then mounts, respawns, and takes the recycled pty id. Client-side correlation from the same run, two controls and one positive in one run differing in exactly one variable: | Tab | Close events observed | Resurrected? | |---|---|---| | `6305cc07` | `user` + `pty-exit` | No | | `ed56f66c` | `user` + `pty-exit` | No | | `2036e760` | `user` only | **YES** | ## The fix `src/shared/closed-terminal-tab-tombstones.ts` (99 lines). A client-recorded close is first-party intent and must survive until the host acknowledges it. Per `docs/reference/ssh-execution-boundary.md` the remote verdict is `unverifiable` -- which may not authorise declaring the process dead, but equally must not authorise resurrecting the tab. This is SSH-v3 principle P2, "durable tombstones with a monotonic per-scope revision", reusing the existing `RemoteWorkspaceSnapshot.revision` rather than adding a twelfth per-tab identity field (the codebase carries eleven, 784 refs, that SSH-v3 Phase 3 deletes). - **Recorded** only on `closeReason === 'user'` (`terminal-tab-close.ts:69`). - **Suppresses** a host-sourced tab only when `tabId in tombstones && !currentTabsById.has(tabId)` -- a live local tab always wins, because deleting a live pane is the one outcome the merge exists to avoid. - **Retires** on positive acknowledgement: `!hostKnownTabIds.has(tabId) && hostRevision > observed`. Strictly newer, so a pull already in flight at close time cannot ack a close it predates. - Three never-retire guards: no revision retires nothing; a worktree the snapshot has no row for retires nothing; the first omitting snapshot only stamps the watermark. - TTL (30d) + cap (500) are **backstops** for a target the user never returns to, not the mechanism. - **Client-local only** -- never crosses the wire, so there is no mixed-version exposure. - Suppression is scoped to `replaceWorktreeIds`, which is what makes the live-tab check meaningful. A final whole-map sweep over the assembled `tabsByWorktree` would break that (a live tab is absent from `currentTabsById` outside the scope and would look suppressible); it is deliberately not there, and the comment at the top of the function says so. ## Evidence The load-bearing evidence is an A/B control on one tree, not the oracle's assertion. Flipping `isSuppressedByClose` to `false` -- one character -- reproduces the resurrection on demand: --repeat-each=2: 1) drop 2 resurrected the closed tab ab0e305d-…: baseline=1 drop1=1 drop2=2 2) drop 2 resurrected the closed tab 51533e34-…: baseline=1 drop1=1 drop2=2 2 failed With suppression on: **0 occurrences of "resurrected the closed tab" across five runs plus one independent run by a second agent.** Provenance verified positively, not by mtime: `closedTerminalTabTombstonesByTabId` appears 13x across 3 renderer chunks including `store-Do3KBvRE.js`; for every red control run `mayCreate` appeared 0 times in `out/main/index.js`. At the unit layer, disabling the same predicate: 3 failed | 39 passed. Restored: 42 passed; 287 across the workspace-session, terminal-store, remote-workspace, shared-tombstone and profile suites; 24 in the four tombstone suites. ## The oracle spec: GREEN in the full lane `ssh-lost-kill-tab-resurrection.spec.ts` passes both tests at this commit. Full Docker-SSH lane, clean tree: BUILD_SHA=49bb96e0b4c DIRTY=0 PROVENANCE tombstone=13 hasLocalTabsRow=2 hostAuthority=4 mayCreate=3 14 specs / 20 tests -> 17 passed, 2 failed, 1 skipped (10.7m) [12/20] :178 does not resurrect tabs whose kill was lost to a killed relay daemon PASSED [13/20] :190 does not resurrect tabs closed while the host is disconnected PASSED grep -c "resurrected the closed tab" (whole lane) -> 0 It passes WITHOUT PR 7 in the build (`mayCreate` present, `SshPtyAbsentFromRelayError` absent), so the bug-2 fix below is not required for it. Test 1 fails intermittently in ISOLATED single-spec runs, where a third defect blocks its cycle-2 setup. The resurrection assertion itself has never failed with this fix in place -- the intermittent failure is always a setup failure, never a resurrected tab. A reviewer running the spec alone may see it red; that is not this fix regressing. Three defects sit under STA-3374 and should not be conflated: - Bug 1 -- the closed tab resurrects. Fixed here. - Bug 2 -- `ssh-pty-session-reattach.ts:227-231` rewrites the relay's `PTY "pty-1" not found` into a bare `SSH_SESSION_EXPIRED`, so `isPtyAlreadyGoneError`'s `/PTY ".+" not found/` cannot match and `attachStablePaneOwner:242`'s already-correct fallback never runs. Owned by PR 7 (`nwparker/ssh-07-absent-from-relay`). Not required for the oracle above. - Bug 3 -- after the daemon is killed and the client launches a replacement, the client's OWN SSH transport drops and does not reconnect within 60s: no "delay step 2/9", no handshake failure, nothing. `ssh-connection.ts:1533` only logs on an SSH-level close. Unfixed, its own ticket. This is what makes test 1 intermittent in isolation. Discriminator for bug 3, measured in the isolated runs (the lane above ran without `ORCA_E2E_FORWARD_APP_LOGS=1`, so it was not re-confirmed there): `[ssh-relay] Socket probe result:` reads "DEAD" on every cycle of test 1 (daemon killed, a NEW relay must be launched) and "ALIVE" on every cycle of test 2 (daemon survived). Whenever a new daemon must be launched, the SSH transport drops afterwards and does not recover. An earlier reading blamed `kill.ts:82-84` for skipping `finishPtyShutdown` on a non-already-gone error. That was eliminated by direct test: the implied fix, `markSshRemotePtyLease(…, 'expired')` in that branch, was implemented, changed nothing, and was reverted rather than shipped unproven. Recorded so the path is not re-walked. The `SSH_SESSION_EXPIRED` rejection is real but fires during cycle 1 for the baseline pane, after which cycle 1 completes; the 60s silence begins only after `Relay channel lost ..., triggering reconnect`. The spec is claimed by the Docker-SSH lane, and that lane does not gate merges today. ## Persistence: the tombstone must survive a relaunch `closedTerminalTabTombstonesByTabId` is declared on `WorkspaceSessionState` but was missing from `workspaceSessionStateSchema` (`src/shared/workspace-session-schema.ts`), which is the load boundary for BOTH partitions -- `normalize-loaded-state-collections.ts` for `local` and `workspace-session-partitions.ts` for `ssh:<target>`. Zod strips unknown keys and the write side does not validate, so the map reached disk and was discarded on the next launch. Measured with the repo's own parser: input : closedTerminalTabTombstonesByTabId: { 'tab-1': {...} } ok = true tombstones after parse = undefined That made the fix ineffective in the exact reported scenario: close an SSH tab with the transport down, QUIT, relaunch, reconnect -- the merge runs with an empty map, the host still lists the tab, and it resurrects. "Every day I open orca and it opens more tabs" is a claim about restarts. Neither the green oracle nor the A/B control could see it: both run entirely inside one app process. It also made the 30-day TTL and the 500 cap unreachable. Fixed by adding the field with a `salvagingRecord` matching its sibling `terminalSurfaceTombstonesByPaneKey`, so one malformed entry drops that entry rather than the map. `workspace-session-schema.ts` was one line under its 300-line max-lines limit, so adding the field required room rather than a suppression (the project forbids max-lines disables and per-file bumps). Two value schemas were extracted to modules named after what they contain: `terminal-tab-id-schema.ts` and `terminal-surface-tombstone-schema.ts`. The closed-tab tombstone's own schema is colocated with its type in `closed-terminal-tab-tombstones.ts`, which is where it belongs -- omitting it from the session schema is exactly the drift that caused this bug. `workspace-session-schema-field-coverage.test.ts` is the ratchet. Two sibling tables already pin themselves with `satisfies Record<keyof WorkspaceSessionState, ...>`; this schema had no such guard and is the one that fell behind. The new file adds both halves -- a `satisfies` list that makes a forgotten field a compile error, and a runtime assertion that names it -- plus a `parseWorkspaceSession` round-trip. Without the schema entry: 3 failed. With it: 3 passed. ## A host tab the user never closed could be deleted `tabId in closedTerminalTabTombstonesByTabId` answers true for every `Object.prototype` key even on an EMPTY map, because the map is a plain object from `Object.fromEntries`. A host tab whose id is `toString` was filtered from the reconciled list, blocked from the host-unknown branch, and stripped of its layout and session id. Tab ids are validated only as non-empty and colon-free, and `createTab` honours caller-supplied id hints, so the id is reachable rather than theoretical. This was the only path in either direction that could delete a tab the user never closed. Now `Object.hasOwn`, as the same file already uses elsewhere. Suppression is also scoped structurally: `isSuppressedByClose` compares the tombstone's stored `worktreeId`, which it already carried, so it cannot reach another workspace's tab. The two sweeps that have no worktree in scope (`terminalLayoutsByTabId`, `remoteSessionIdsByTabId`) now consult the set of ids this merge actually suppressed rather than re-deriving a verdict without that scope. The scope comment at the top of the function was also wrong and is corrected. It claimed every use of suppression sits inside `replaceWorktreeIds`; it does not -- the tabs pass walks all of `orderedWorktreeIds` and the two sweeps cover the whole remote maps. What actually makes it safe is that `closeTab` strips the id from every worktree row before recording the tombstone, plus the worktree match above, plus `closeReason === 'user'` being the only writer. Real guarantee, different from the documented one. ## Divergences from open PR #16571 #16571 implements the same concept. Three deliberate changes: 1. It never retires on acknowledgement -- TTL+cap only, so it never converges. Ack retirement added. 2. It crosses the wire and lets a HOST-sourced tombstone delete a LOCAL tab in a final whole-map sweep. After #14361 that is the wrong risk; dropped. This also removes the mixed-version regression its own body flags. 3. Its hydration unions rather than replaces the map -- a union resurrects every tombstone the merge just retired, so it never converges. Its `activeTabId` nulling is also dropped as redundant: `workspace-terminal-hydration.ts:99-105,126-138` already revalidates both pointers against the tab rows it just built, and nulling twice would add a second rule that has to stay in step with the first. ## Can a tab the user did NOT close disappear? No, but the guarantee needs stating precisely. The only writer is `recordClosedTerminalTabTombstone` (`terminal-tab-close.ts:69`), reachable only on `closeReason === 'user'`; suppression additionally requires the tab not be live locally. Reopen (`recently-closed-tabs.ts:122-166`) calls `createTab` and restores cwd/shell/title/color/position, never the old id. **Caveat, stated because the slogan is not literally true:** `createTab` honours a caller-supplied id hint (`terminal-tab-creation.ts:53-65`, used by `useIpcEvents` for host-admitted tabs), so "tab ids are uuids that never recur" does not hold in this codebase. The guarantee rests on the `closeReason === 'user'` writer plus the live-local-tab check, not on id uniqueness. ## Risk Renderer-side, client-local, no wire change. The blast radius is `mergeDirectSshRemoteWorkspaceSession` and the persisted session field. Worst case if the ack logic were wrong in the retiring direction: a tombstone outlives its usefulness and suppresses a host tab whose id the host re-issues -- bounded by the live-local-tab check, the 30d TTL and the 500 cap. Worst case in the other direction is today's behaviour. `profile-project-session-field-disposition.ts` records the new field as `notRepoScoped` / `notTransferred` residue, bounded by the same TTL and cap. ## Verify pnpm test src/shared/closed-terminal-tab-tombstones.test.ts \ src/renderer/src/lib/workspace-session-closed-tab-tombstones.test.ts \ src/renderer/src/store/terminals/terminal-tab-close-tombstone.test.ts \ src/renderer/src/hooks/remote-workspace-session-merge-close-tombstones.test.ts To reproduce the bug this fixes, set `isSuppressedByClose` to `() => false` in `remote-workspace-session-merge.ts` and run `pnpm test:e2e:ssh-docker -- tests/e2e/ssh-lost-kill-tab-resurrection.spec.ts --repeat-each=2`.
An enterprise user: "Every day I open orca and it opens more tabs daily at a linear scale." Three reports over a week, told on 08-19 that a PR had fixed it, reported twice more after. STA-4658 (P0), GH #12447, #15136, #10342, #9585. One install held 39 zombie tab records. The revived tab's sleeping-agent record still holds the pre-close session id, so it boots `claude --resume <old id>` -- two agents on one transcript. ## The chain, measured Reproduced deterministically in `ssh-lost-kill-tab-resurrection.spec.ts`: close an SSH tab, kill the relay daemon in the container so `pty.kill` rejects with a transport-class error, reconnect. drop 2 resurrected the closed tab <id>: baseline=1 drop1=1 drop2=2 (closed tab returned) drop3=1 The trigger is narrow and had to be measured rather than assumed: killed relay daemon reproduces **6 of 6 runs**; an orderly `ssh.disconnect` **passes**. Only an ungraceful loss -- network partition, host reboot, relay crash, a laptop sleeping mid-session -- strands the close with the RPC rejecting on a transport-class error. Both variants live in the spec behind one `runResurrectionCycles` parameterized solely by the disruption, so the difference is attributable to that single variable. What actually carries the tab back, from the pull path (`workspace.get` -> `getRemoteSnapshot`, `remote-workspace-relay-sync.ts:29`): pullSnapshot rev=3 tabs={repo:["16c4a3e1","06aba6b6"]} pullSnapshot rev=4 tabs={repo:["16c4a3e1","ff72768e"]} <- ff72768e IS the resurrected tab pullSnapshot rev=5 tabs={repo:["16c4a3e1","ff72768e","da21b76c"]} The client uploaded the session containing the tab; the user closed it; the kill RPC rejected so the close never reached the host; the host's snapshot still lists it; the client pulls it back and the merge restores it -- **correctly, by its own rule that the host is authoritative for what it knows.** A pane then mounts, respawns, and takes the recycled pty id. Client-side correlation from the same run, two controls and one positive in one run differing in exactly one variable: | Tab | Close events observed | Resurrected? | |---|---|---| | `6305cc07` | `user` + `pty-exit` | No | | `ed56f66c` | `user` + `pty-exit` | No | | `2036e760` | `user` only | **YES** | ## The fix `src/shared/closed-terminal-tab-tombstones.ts` (99 lines). A client-recorded close is first-party intent and must survive until the host acknowledges it. Per `docs/reference/ssh-execution-boundary.md` the remote verdict is `unverifiable` -- which may not authorise declaring the process dead, but equally must not authorise resurrecting the tab. This is SSH-v3 principle P2, "durable tombstones with a monotonic per-scope revision", reusing the existing `RemoteWorkspaceSnapshot.revision` rather than adding a twelfth per-tab identity field (the codebase carries eleven, 784 refs, that SSH-v3 Phase 3 deletes). - **Recorded** only on `closeReason === 'user'` (`terminal-tab-close.ts:69`). - **Suppresses** a host-sourced tab only when `tabId in tombstones && !currentTabsById.has(tabId)` -- a live local tab always wins, because deleting a live pane is the one outcome the merge exists to avoid. - **Retires** on positive acknowledgement: `!hostKnownTabIds.has(tabId) && hostRevision > observed`. Strictly newer, so a pull already in flight at close time cannot ack a close it predates. - Three never-retire guards: no revision retires nothing; a worktree the snapshot has no row for retires nothing; the first omitting snapshot only stamps the watermark. - TTL (30d) + cap (500) are **backstops** for a target the user never returns to, not the mechanism. - **Client-local only** -- never crosses the wire, so there is no mixed-version exposure. - Suppression is scoped to `replaceWorktreeIds`, which is what makes the live-tab check meaningful. A final whole-map sweep over the assembled `tabsByWorktree` would break that (a live tab is absent from `currentTabsById` outside the scope and would look suppressible); it is deliberately not there, and the comment at the top of the function says so. ## Evidence The load-bearing evidence is an A/B control on one tree, not the oracle's assertion. Flipping `isSuppressedByClose` to `false` -- one character -- reproduces the resurrection on demand: --repeat-each=2: 1) drop 2 resurrected the closed tab ab0e305d-…: baseline=1 drop1=1 drop2=2 2) drop 2 resurrected the closed tab 51533e34-…: baseline=1 drop1=1 drop2=2 2 failed With suppression on: **0 occurrences of "resurrected the closed tab" across five runs plus one independent run by a second agent.** Provenance verified positively, not by mtime: `closedTerminalTabTombstonesByTabId` appears 13x across 3 renderer chunks including `store-Do3KBvRE.js`; for every red control run `mayCreate` appeared 0 times in `out/main/index.js`. At the unit layer, disabling the same predicate: 3 failed | 39 passed. Restored: 42 passed; 287 across the workspace-session, terminal-store, remote-workspace, shared-tombstone and profile suites; 24 in the four tombstone suites. ## The oracle spec: GREEN in the full lane `ssh-lost-kill-tab-resurrection.spec.ts` passes both tests at this commit. Full Docker-SSH lane, clean tree: BUILD_SHA=49bb96e0b4c DIRTY=0 PROVENANCE tombstone=13 hasLocalTabsRow=2 hostAuthority=4 mayCreate=3 14 specs / 20 tests -> 17 passed, 2 failed, 1 skipped (10.7m) [12/20] :178 does not resurrect tabs whose kill was lost to a killed relay daemon PASSED [13/20] :190 does not resurrect tabs closed while the host is disconnected PASSED grep -c "resurrected the closed tab" (whole lane) -> 0 It passes WITHOUT PR 7 in the build (`mayCreate` present, `SshPtyAbsentFromRelayError` absent), so the bug-2 fix below is not required for it. Test 1 fails intermittently in ISOLATED single-spec runs, where a third defect blocks its cycle-2 setup. The resurrection assertion itself has never failed with this fix in place -- the intermittent failure is always a setup failure, never a resurrected tab. A reviewer running the spec alone may see it red; that is not this fix regressing. Three defects sit under STA-3374 and should not be conflated: - Bug 1 -- the closed tab resurrects. Fixed here. - Bug 2 -- `ssh-pty-session-reattach.ts:227-231` rewrites the relay's `PTY "pty-1" not found` into a bare `SSH_SESSION_EXPIRED`, so `isPtyAlreadyGoneError`'s `/PTY ".+" not found/` cannot match and `attachStablePaneOwner:242`'s already-correct fallback never runs. Owned by PR 7 (`nwparker/ssh-07-absent-from-relay`). Not required for the oracle above. - Bug 3 -- after the daemon is killed and the client launches a replacement, the client's OWN SSH transport drops and does not reconnect within 60s: no "delay step 2/9", no handshake failure, nothing. `ssh-connection.ts:1533` only logs on an SSH-level close. Unfixed, its own ticket. This is what makes test 1 intermittent in isolation. Discriminator for bug 3, measured in the isolated runs (the lane above ran without `ORCA_E2E_FORWARD_APP_LOGS=1`, so it was not re-confirmed there): `[ssh-relay] Socket probe result:` reads "DEAD" on every cycle of test 1 (daemon killed, a NEW relay must be launched) and "ALIVE" on every cycle of test 2 (daemon survived). Whenever a new daemon must be launched, the SSH transport drops afterwards and does not recover. An earlier reading blamed `kill.ts:82-84` for skipping `finishPtyShutdown` on a non-already-gone error. That was eliminated by direct test: the implied fix, `markSshRemotePtyLease(…, 'expired')` in that branch, was implemented, changed nothing, and was reverted rather than shipped unproven. Recorded so the path is not re-walked. The `SSH_SESSION_EXPIRED` rejection is real but fires during cycle 1 for the baseline pane, after which cycle 1 completes; the 60s silence begins only after `Relay channel lost ..., triggering reconnect`. The spec is claimed by the Docker-SSH lane, and that lane does not gate merges today. ## Persistence: the tombstone must survive a relaunch `closedTerminalTabTombstonesByTabId` is declared on `WorkspaceSessionState` but was missing from `workspaceSessionStateSchema` (`src/shared/workspace-session-schema.ts`), which is the load boundary for BOTH partitions -- `normalize-loaded-state-collections.ts` for `local` and `workspace-session-partitions.ts` for `ssh:<target>`. Zod strips unknown keys and the write side does not validate, so the map reached disk and was discarded on the next launch. Measured with the repo's own parser: input : closedTerminalTabTombstonesByTabId: { 'tab-1': {...} } ok = true tombstones after parse = undefined That made the fix ineffective in the exact reported scenario: close an SSH tab with the transport down, QUIT, relaunch, reconnect -- the merge runs with an empty map, the host still lists the tab, and it resurrects. "Every day I open orca and it opens more tabs" is a claim about restarts. Neither the green oracle nor the A/B control could see it: both run entirely inside one app process. It also made the 30-day TTL and the 500 cap unreachable. Fixed by adding the field with a `salvagingRecord` matching its sibling `terminalSurfaceTombstonesByPaneKey`, so one malformed entry drops that entry rather than the map. `workspace-session-schema.ts` was one line under its 300-line max-lines limit, so adding the field required room rather than a suppression (the project forbids max-lines disables and per-file bumps). Two value schemas were extracted to modules named after what they contain: `terminal-tab-id-schema.ts` and `terminal-surface-tombstone-schema.ts`. The closed-tab tombstone's own schema is colocated with its type in `closed-terminal-tab-tombstones.ts`, which is where it belongs -- omitting it from the session schema is exactly the drift that caused this bug. `workspace-session-schema-field-coverage.test.ts` is the ratchet. Two sibling tables already pin themselves with `satisfies Record<keyof WorkspaceSessionState, ...>`; this schema had no such guard and is the one that fell behind. The new file adds both halves -- a `satisfies` list that makes a forgotten field a compile error, and a runtime assertion that names it -- plus a `parseWorkspaceSession` round-trip. Without the schema entry: 3 failed. With it: 3 passed. ## A host tab the user never closed could be deleted `tabId in closedTerminalTabTombstonesByTabId` answers true for every `Object.prototype` key even on an EMPTY map, because the map is a plain object from `Object.fromEntries`. A host tab whose id is `toString` was filtered from the reconciled list, blocked from the host-unknown branch, and stripped of its layout and session id. Tab ids are validated only as non-empty and colon-free, and `createTab` honours caller-supplied id hints, so the id is reachable rather than theoretical. This was the only path in either direction that could delete a tab the user never closed. Now `Object.hasOwn`, as the same file already uses elsewhere. Suppression is also scoped structurally: `isSuppressedByClose` compares the tombstone's stored `worktreeId`, which it already carried, so it cannot reach another workspace's tab. The two sweeps that have no worktree in scope (`terminalLayoutsByTabId`, `remoteSessionIdsByTabId`) now consult the set of ids this merge actually suppressed rather than re-deriving a verdict without that scope. The scope comment at the top of the function was also wrong and is corrected. It claimed every use of suppression sits inside `replaceWorktreeIds`; it does not -- the tabs pass walks all of `orderedWorktreeIds` and the two sweeps cover the whole remote maps. What actually makes it safe is that `closeTab` strips the id from every worktree row before recording the tombstone, plus the worktree match above, plus `closeReason === 'user'` being the only writer. Real guarantee, different from the documented one. ## Divergences from open PR #16571 #16571 implements the same concept. Three deliberate changes: 1. It never retires on acknowledgement -- TTL+cap only, so it never converges. Ack retirement added. 2. It crosses the wire and lets a HOST-sourced tombstone delete a LOCAL tab in a final whole-map sweep. After #14361 that is the wrong risk; dropped. This also removes the mixed-version regression its own body flags. 3. Its hydration unions rather than replaces the map -- a union resurrects every tombstone the merge just retired, so it never converges. Its `activeTabId` nulling is also dropped as redundant: `workspace-terminal-hydration.ts:99-105,126-138` already revalidates both pointers against the tab rows it just built, and nulling twice would add a second rule that has to stay in step with the first. ## Can a tab the user did NOT close disappear? No, but the guarantee needs stating precisely. The only writer is `recordClosedTerminalTabTombstone` (`terminal-tab-close.ts:69`), reachable only on `closeReason === 'user'`; suppression additionally requires the tab not be live locally. Reopen (`recently-closed-tabs.ts:122-166`) calls `createTab` and restores cwd/shell/title/color/position, never the old id. **Caveat, stated because the slogan is not literally true:** `createTab` honours a caller-supplied id hint (`terminal-tab-creation.ts:53-65`, used by `useIpcEvents` for host-admitted tabs), so "tab ids are uuids that never recur" does not hold in this codebase. The guarantee rests on the `closeReason === 'user'` writer plus the live-local-tab check, not on id uniqueness. ## Risk Renderer-side, client-local, no wire change. The blast radius is `mergeDirectSshRemoteWorkspaceSession` and the persisted session field. Worst case if the ack logic were wrong in the retiring direction: a tombstone outlives its usefulness and suppresses a host tab whose id the host re-issues -- bounded by the live-local-tab check, the 30d TTL and the 500 cap. Worst case in the other direction is today's behaviour. `profile-project-session-field-disposition.ts` records the new field as `notRepoScoped` / `notTransferred` residue, bounded by the same TTL and cap. ## Verify pnpm test src/shared/closed-terminal-tab-tombstones.test.ts \ src/renderer/src/lib/workspace-session-closed-tab-tombstones.test.ts \ src/renderer/src/store/terminals/terminal-tab-close-tombstone.test.ts \ src/renderer/src/hooks/remote-workspace-session-merge-close-tombstones.test.ts To reproduce the bug this fixes, set `isSuppressedByClose` to `() => false` in `remote-workspace-session-merge.ts` and run `pnpm test:e2e:ssh-docker -- tests/e2e/ssh-lost-kill-tab-resurrection.spec.ts --repeat-each=2`.
#16752) An enterprise user: "Every day I open orca and it opens more tabs daily at a linear scale." Three reports over a week, told on 08-19 that a PR had fixed it, reported twice more after. STA-4658 (P0), GH #12447, #15136, #10342, #9585. One install held 39 zombie tab records. The revived tab's sleeping-agent record still holds the pre-close session id, so it boots `claude --resume <old id>` -- two agents on one transcript. ## The chain, measured Reproduced deterministically in `ssh-lost-kill-tab-resurrection.spec.ts`: close an SSH tab, kill the relay daemon in the container so `pty.kill` rejects with a transport-class error, reconnect. drop 2 resurrected the closed tab <id>: baseline=1 drop1=1 drop2=2 (closed tab returned) drop3=1 The trigger is narrow and had to be measured rather than assumed: killed relay daemon reproduces **6 of 6 runs**; an orderly `ssh.disconnect` **passes**. Only an ungraceful loss -- network partition, host reboot, relay crash, a laptop sleeping mid-session -- strands the close with the RPC rejecting on a transport-class error. Both variants live in the spec behind one `runResurrectionCycles` parameterized solely by the disruption, so the difference is attributable to that single variable. What actually carries the tab back, from the pull path (`workspace.get` -> `getRemoteSnapshot`, `remote-workspace-relay-sync.ts:29`): pullSnapshot rev=3 tabs={repo:["16c4a3e1","06aba6b6"]} pullSnapshot rev=4 tabs={repo:["16c4a3e1","ff72768e"]} <- ff72768e IS the resurrected tab pullSnapshot rev=5 tabs={repo:["16c4a3e1","ff72768e","da21b76c"]} The client uploaded the session containing the tab; the user closed it; the kill RPC rejected so the close never reached the host; the host's snapshot still lists it; the client pulls it back and the merge restores it -- **correctly, by its own rule that the host is authoritative for what it knows.** A pane then mounts, respawns, and takes the recycled pty id. Client-side correlation from the same run, two controls and one positive in one run differing in exactly one variable: | Tab | Close events observed | Resurrected? | |---|---|---| | `6305cc07` | `user` + `pty-exit` | No | | `ed56f66c` | `user` + `pty-exit` | No | | `2036e760` | `user` only | **YES** | ## The fix `src/shared/closed-terminal-tab-tombstones.ts` (99 lines). A client-recorded close is first-party intent and must survive until the host acknowledges it. Per `docs/reference/ssh-execution-boundary.md` the remote verdict is `unverifiable` -- which may not authorise declaring the process dead, but equally must not authorise resurrecting the tab. This is SSH-v3 principle P2, "durable tombstones with a monotonic per-scope revision", reusing the existing `RemoteWorkspaceSnapshot.revision` rather than adding a twelfth per-tab identity field (the codebase carries eleven, 784 refs, that SSH-v3 Phase 3 deletes). - **Recorded** only on `closeReason === 'user'` (`terminal-tab-close.ts:69`). - **Suppresses** a host-sourced tab only when `tabId in tombstones && !currentTabsById.has(tabId)` -- a live local tab always wins, because deleting a live pane is the one outcome the merge exists to avoid. - **Retires** on positive acknowledgement: `!hostKnownTabIds.has(tabId) && hostRevision > observed`. Strictly newer, so a pull already in flight at close time cannot ack a close it predates. - Three never-retire guards: no revision retires nothing; a worktree the snapshot has no row for retires nothing; the first omitting snapshot only stamps the watermark. - TTL (30d) + cap (500) are **backstops** for a target the user never returns to, not the mechanism. - **Client-local only** -- never crosses the wire, so there is no mixed-version exposure. - Suppression is scoped to `replaceWorktreeIds`, which is what makes the live-tab check meaningful. A final whole-map sweep over the assembled `tabsByWorktree` would break that (a live tab is absent from `currentTabsById` outside the scope and would look suppressible); it is deliberately not there, and the comment at the top of the function says so. ## Evidence The load-bearing evidence is an A/B control on one tree, not the oracle's assertion. Flipping `isSuppressedByClose` to `false` -- one character -- reproduces the resurrection on demand: --repeat-each=2: 1) drop 2 resurrected the closed tab ab0e305d-…: baseline=1 drop1=1 drop2=2 2) drop 2 resurrected the closed tab 51533e34-…: baseline=1 drop1=1 drop2=2 2 failed With suppression on: **0 occurrences of "resurrected the closed tab" across five runs plus one independent run by a second agent.** Provenance verified positively, not by mtime: `closedTerminalTabTombstonesByTabId` appears 13x across 3 renderer chunks including `store-Do3KBvRE.js`; for every red control run `mayCreate` appeared 0 times in `out/main/index.js`. At the unit layer, disabling the same predicate: 3 failed | 39 passed. Restored: 42 passed; 287 across the workspace-session, terminal-store, remote-workspace, shared-tombstone and profile suites; 24 in the four tombstone suites. ## The oracle spec: GREEN in the full lane `ssh-lost-kill-tab-resurrection.spec.ts` passes both tests at this commit. Full Docker-SSH lane, clean tree: BUILD_SHA=49bb96e0b4c DIRTY=0 PROVENANCE tombstone=13 hasLocalTabsRow=2 hostAuthority=4 mayCreate=3 14 specs / 20 tests -> 17 passed, 2 failed, 1 skipped (10.7m) [12/20] :178 does not resurrect tabs whose kill was lost to a killed relay daemon PASSED [13/20] :190 does not resurrect tabs closed while the host is disconnected PASSED grep -c "resurrected the closed tab" (whole lane) -> 0 It passes WITHOUT PR 7 in the build (`mayCreate` present, `SshPtyAbsentFromRelayError` absent), so the bug-2 fix below is not required for it. Test 1 fails intermittently in ISOLATED single-spec runs, where a third defect blocks its cycle-2 setup. The resurrection assertion itself has never failed with this fix in place -- the intermittent failure is always a setup failure, never a resurrected tab. A reviewer running the spec alone may see it red; that is not this fix regressing. Three defects sit under STA-3374 and should not be conflated: - Bug 1 -- the closed tab resurrects. Fixed here. - Bug 2 -- `ssh-pty-session-reattach.ts:227-231` rewrites the relay's `PTY "pty-1" not found` into a bare `SSH_SESSION_EXPIRED`, so `isPtyAlreadyGoneError`'s `/PTY ".+" not found/` cannot match and `attachStablePaneOwner:242`'s already-correct fallback never runs. Owned by PR 7 (`nwparker/ssh-07-absent-from-relay`). Not required for the oracle above. - Bug 3 -- after the daemon is killed and the client launches a replacement, the client's OWN SSH transport drops and does not reconnect within 60s: no "delay step 2/9", no handshake failure, nothing. `ssh-connection.ts:1533` only logs on an SSH-level close. Unfixed, its own ticket. This is what makes test 1 intermittent in isolation. Discriminator for bug 3, measured in the isolated runs (the lane above ran without `ORCA_E2E_FORWARD_APP_LOGS=1`, so it was not re-confirmed there): `[ssh-relay] Socket probe result:` reads "DEAD" on every cycle of test 1 (daemon killed, a NEW relay must be launched) and "ALIVE" on every cycle of test 2 (daemon survived). Whenever a new daemon must be launched, the SSH transport drops afterwards and does not recover. An earlier reading blamed `kill.ts:82-84` for skipping `finishPtyShutdown` on a non-already-gone error. That was eliminated by direct test: the implied fix, `markSshRemotePtyLease(…, 'expired')` in that branch, was implemented, changed nothing, and was reverted rather than shipped unproven. Recorded so the path is not re-walked. The `SSH_SESSION_EXPIRED` rejection is real but fires during cycle 1 for the baseline pane, after which cycle 1 completes; the 60s silence begins only after `Relay channel lost ..., triggering reconnect`. The spec is claimed by the Docker-SSH lane, and that lane does not gate merges today. ## Persistence: the tombstone must survive a relaunch `closedTerminalTabTombstonesByTabId` is declared on `WorkspaceSessionState` but was missing from `workspaceSessionStateSchema` (`src/shared/workspace-session-schema.ts`), which is the load boundary for BOTH partitions -- `normalize-loaded-state-collections.ts` for `local` and `workspace-session-partitions.ts` for `ssh:<target>`. Zod strips unknown keys and the write side does not validate, so the map reached disk and was discarded on the next launch. Measured with the repo's own parser: input : closedTerminalTabTombstonesByTabId: { 'tab-1': {...} } ok = true tombstones after parse = undefined That made the fix ineffective in the exact reported scenario: close an SSH tab with the transport down, QUIT, relaunch, reconnect -- the merge runs with an empty map, the host still lists the tab, and it resurrects. "Every day I open orca and it opens more tabs" is a claim about restarts. Neither the green oracle nor the A/B control could see it: both run entirely inside one app process. It also made the 30-day TTL and the 500 cap unreachable. Fixed by adding the field with a `salvagingRecord` matching its sibling `terminalSurfaceTombstonesByPaneKey`, so one malformed entry drops that entry rather than the map. `workspace-session-schema.ts` was one line under its 300-line max-lines limit, so adding the field required room rather than a suppression (the project forbids max-lines disables and per-file bumps). Two value schemas were extracted to modules named after what they contain: `terminal-tab-id-schema.ts` and `terminal-surface-tombstone-schema.ts`. The closed-tab tombstone's own schema is colocated with its type in `closed-terminal-tab-tombstones.ts`, which is where it belongs -- omitting it from the session schema is exactly the drift that caused this bug. `workspace-session-schema-field-coverage.test.ts` is the ratchet. Two sibling tables already pin themselves with `satisfies Record<keyof WorkspaceSessionState, ...>`; this schema had no such guard and is the one that fell behind. The new file adds both halves -- a `satisfies` list that makes a forgotten field a compile error, and a runtime assertion that names it -- plus a `parseWorkspaceSession` round-trip. Without the schema entry: 3 failed. With it: 3 passed. ## A host tab the user never closed could be deleted `tabId in closedTerminalTabTombstonesByTabId` answers true for every `Object.prototype` key even on an EMPTY map, because the map is a plain object from `Object.fromEntries`. A host tab whose id is `toString` was filtered from the reconciled list, blocked from the host-unknown branch, and stripped of its layout and session id. Tab ids are validated only as non-empty and colon-free, and `createTab` honours caller-supplied id hints, so the id is reachable rather than theoretical. This was the only path in either direction that could delete a tab the user never closed. Now `Object.hasOwn`, as the same file already uses elsewhere. Suppression is also scoped structurally: `isSuppressedByClose` compares the tombstone's stored `worktreeId`, which it already carried, so it cannot reach another workspace's tab. The two sweeps that have no worktree in scope (`terminalLayoutsByTabId`, `remoteSessionIdsByTabId`) now consult the set of ids this merge actually suppressed rather than re-deriving a verdict without that scope. The scope comment at the top of the function was also wrong and is corrected. It claimed every use of suppression sits inside `replaceWorktreeIds`; it does not -- the tabs pass walks all of `orderedWorktreeIds` and the two sweeps cover the whole remote maps. What actually makes it safe is that `closeTab` strips the id from every worktree row before recording the tombstone, plus the worktree match above, plus `closeReason === 'user'` being the only writer. Real guarantee, different from the documented one. ## Divergences from open PR #16571 #16571 implements the same concept. Three deliberate changes: 1. It never retires on acknowledgement -- TTL+cap only, so it never converges. Ack retirement added. 2. It crosses the wire and lets a HOST-sourced tombstone delete a LOCAL tab in a final whole-map sweep. After #14361 that is the wrong risk; dropped. This also removes the mixed-version regression its own body flags. 3. Its hydration unions rather than replaces the map -- a union resurrects every tombstone the merge just retired, so it never converges. Its `activeTabId` nulling is also dropped as redundant: `workspace-terminal-hydration.ts:99-105,126-138` already revalidates both pointers against the tab rows it just built, and nulling twice would add a second rule that has to stay in step with the first. ## Can a tab the user did NOT close disappear? No, but the guarantee needs stating precisely. The only writer is `recordClosedTerminalTabTombstone` (`terminal-tab-close.ts:69`), reachable only on `closeReason === 'user'`; suppression additionally requires the tab not be live locally. Reopen (`recently-closed-tabs.ts:122-166`) calls `createTab` and restores cwd/shell/title/color/position, never the old id. **Caveat, stated because the slogan is not literally true:** `createTab` honours a caller-supplied id hint (`terminal-tab-creation.ts:53-65`, used by `useIpcEvents` for host-admitted tabs), so "tab ids are uuids that never recur" does not hold in this codebase. The guarantee rests on the `closeReason === 'user'` writer plus the live-local-tab check, not on id uniqueness. ## Risk Renderer-side, client-local, no wire change. The blast radius is `mergeDirectSshRemoteWorkspaceSession` and the persisted session field. Worst case if the ack logic were wrong in the retiring direction: a tombstone outlives its usefulness and suppresses a host tab whose id the host re-issues -- bounded by the live-local-tab check, the 30d TTL and the 500 cap. Worst case in the other direction is today's behaviour. `profile-project-session-field-disposition.ts` records the new field as `notRepoScoped` / `notTransferred` residue, bounded by the same TTL and cap. ## Verify pnpm test src/shared/closed-terminal-tab-tombstones.test.ts \ src/renderer/src/lib/workspace-session-closed-tab-tombstones.test.ts \ src/renderer/src/store/terminals/terminal-tab-close-tombstone.test.ts \ src/renderer/src/hooks/remote-workspace-session-merge-close-tombstones.test.ts To reproduce the bug this fixes, set `isSuppressedByClose` to `() => false` in `remote-workspace-session-merge.ts` and run `pnpm test:e2e:ssh-docker -- tests/e2e/ssh-lost-kill-tab-resurrection.spec.ts --repeat-each=2`.
|
Superseded by #16752, which landed the same concept with three deliberate divergences after readiness review:
Also from the review of this concept, both fixed in #16752:
Two smaller things from this PR that did not carry over, deliberately: the Minor: Closing as superseded. Reopen if any of the above is wrong. |
stablyai#16752) An enterprise user: "Every day I open orca and it opens more tabs daily at a linear scale." Three reports over a week, told on 08-19 that a PR had fixed it, reported twice more after. STA-4658 (P0), GH stablyai#12447, stablyai#15136, stablyai#10342, stablyai#9585. One install held 39 zombie tab records. The revived tab's sleeping-agent record still holds the pre-close session id, so it boots `claude --resume <old id>` -- two agents on one transcript. ## The chain, measured Reproduced deterministically in `ssh-lost-kill-tab-resurrection.spec.ts`: close an SSH tab, kill the relay daemon in the container so `pty.kill` rejects with a transport-class error, reconnect. drop 2 resurrected the closed tab <id>: baseline=1 drop1=1 drop2=2 (closed tab returned) drop3=1 The trigger is narrow and had to be measured rather than assumed: killed relay daemon reproduces **6 of 6 runs**; an orderly `ssh.disconnect` **passes**. Only an ungraceful loss -- network partition, host reboot, relay crash, a laptop sleeping mid-session -- strands the close with the RPC rejecting on a transport-class error. Both variants live in the spec behind one `runResurrectionCycles` parameterized solely by the disruption, so the difference is attributable to that single variable. What actually carries the tab back, from the pull path (`workspace.get` -> `getRemoteSnapshot`, `remote-workspace-relay-sync.ts:29`): pullSnapshot rev=3 tabs={repo:["16c4a3e1","06aba6b6"]} pullSnapshot rev=4 tabs={repo:["16c4a3e1","ff72768e"]} <- ff72768e IS the resurrected tab pullSnapshot rev=5 tabs={repo:["16c4a3e1","ff72768e","da21b76c"]} The client uploaded the session containing the tab; the user closed it; the kill RPC rejected so the close never reached the host; the host's snapshot still lists it; the client pulls it back and the merge restores it -- **correctly, by its own rule that the host is authoritative for what it knows.** A pane then mounts, respawns, and takes the recycled pty id. Client-side correlation from the same run, two controls and one positive in one run differing in exactly one variable: | Tab | Close events observed | Resurrected? | |---|---|---| | `6305cc07` | `user` + `pty-exit` | No | | `ed56f66c` | `user` + `pty-exit` | No | | `2036e760` | `user` only | **YES** | ## The fix `src/shared/closed-terminal-tab-tombstones.ts` (99 lines). A client-recorded close is first-party intent and must survive until the host acknowledges it. Per `docs/reference/ssh-execution-boundary.md` the remote verdict is `unverifiable` -- which may not authorise declaring the process dead, but equally must not authorise resurrecting the tab. This is SSH-v3 principle P2, "durable tombstones with a monotonic per-scope revision", reusing the existing `RemoteWorkspaceSnapshot.revision` rather than adding a twelfth per-tab identity field (the codebase carries eleven, 784 refs, that SSH-v3 Phase 3 deletes). - **Recorded** only on `closeReason === 'user'` (`terminal-tab-close.ts:69`). - **Suppresses** a host-sourced tab only when `tabId in tombstones && !currentTabsById.has(tabId)` -- a live local tab always wins, because deleting a live pane is the one outcome the merge exists to avoid. - **Retires** on positive acknowledgement: `!hostKnownTabIds.has(tabId) && hostRevision > observed`. Strictly newer, so a pull already in flight at close time cannot ack a close it predates. - Three never-retire guards: no revision retires nothing; a worktree the snapshot has no row for retires nothing; the first omitting snapshot only stamps the watermark. - TTL (30d) + cap (500) are **backstops** for a target the user never returns to, not the mechanism. - **Client-local only** -- never crosses the wire, so there is no mixed-version exposure. - Suppression is scoped to `replaceWorktreeIds`, which is what makes the live-tab check meaningful. A final whole-map sweep over the assembled `tabsByWorktree` would break that (a live tab is absent from `currentTabsById` outside the scope and would look suppressible); it is deliberately not there, and the comment at the top of the function says so. ## Evidence The load-bearing evidence is an A/B control on one tree, not the oracle's assertion. Flipping `isSuppressedByClose` to `false` -- one character -- reproduces the resurrection on demand: --repeat-each=2: 1) drop 2 resurrected the closed tab ab0e305d-…: baseline=1 drop1=1 drop2=2 2) drop 2 resurrected the closed tab 51533e34-…: baseline=1 drop1=1 drop2=2 2 failed With suppression on: **0 occurrences of "resurrected the closed tab" across five runs plus one independent run by a second agent.** Provenance verified positively, not by mtime: `closedTerminalTabTombstonesByTabId` appears 13x across 3 renderer chunks including `store-Do3KBvRE.js`; for every red control run `mayCreate` appeared 0 times in `out/main/index.js`. At the unit layer, disabling the same predicate: 3 failed | 39 passed. Restored: 42 passed; 287 across the workspace-session, terminal-store, remote-workspace, shared-tombstone and profile suites; 24 in the four tombstone suites. ## The oracle spec: GREEN in the full lane `ssh-lost-kill-tab-resurrection.spec.ts` passes both tests at this commit. Full Docker-SSH lane, clean tree: BUILD_SHA=49bb96e0b4c DIRTY=0 PROVENANCE tombstone=13 hasLocalTabsRow=2 hostAuthority=4 mayCreate=3 14 specs / 20 tests -> 17 passed, 2 failed, 1 skipped (10.7m) [12/20] :178 does not resurrect tabs whose kill was lost to a killed relay daemon PASSED [13/20] :190 does not resurrect tabs closed while the host is disconnected PASSED grep -c "resurrected the closed tab" (whole lane) -> 0 It passes WITHOUT PR 7 in the build (`mayCreate` present, `SshPtyAbsentFromRelayError` absent), so the bug-2 fix below is not required for it. Test 1 fails intermittently in ISOLATED single-spec runs, where a third defect blocks its cycle-2 setup. The resurrection assertion itself has never failed with this fix in place -- the intermittent failure is always a setup failure, never a resurrected tab. A reviewer running the spec alone may see it red; that is not this fix regressing. Three defects sit under STA-3374 and should not be conflated: - Bug 1 -- the closed tab resurrects. Fixed here. - Bug 2 -- `ssh-pty-session-reattach.ts:227-231` rewrites the relay's `PTY "pty-1" not found` into a bare `SSH_SESSION_EXPIRED`, so `isPtyAlreadyGoneError`'s `/PTY ".+" not found/` cannot match and `attachStablePaneOwner:242`'s already-correct fallback never runs. Owned by PR 7 (`nwparker/ssh-07-absent-from-relay`). Not required for the oracle above. - Bug 3 -- after the daemon is killed and the client launches a replacement, the client's OWN SSH transport drops and does not reconnect within 60s: no "delay step 2/9", no handshake failure, nothing. `ssh-connection.ts:1533` only logs on an SSH-level close. Unfixed, its own ticket. This is what makes test 1 intermittent in isolation. Discriminator for bug 3, measured in the isolated runs (the lane above ran without `ORCA_E2E_FORWARD_APP_LOGS=1`, so it was not re-confirmed there): `[ssh-relay] Socket probe result:` reads "DEAD" on every cycle of test 1 (daemon killed, a NEW relay must be launched) and "ALIVE" on every cycle of test 2 (daemon survived). Whenever a new daemon must be launched, the SSH transport drops afterwards and does not recover. An earlier reading blamed `kill.ts:82-84` for skipping `finishPtyShutdown` on a non-already-gone error. That was eliminated by direct test: the implied fix, `markSshRemotePtyLease(…, 'expired')` in that branch, was implemented, changed nothing, and was reverted rather than shipped unproven. Recorded so the path is not re-walked. The `SSH_SESSION_EXPIRED` rejection is real but fires during cycle 1 for the baseline pane, after which cycle 1 completes; the 60s silence begins only after `Relay channel lost ..., triggering reconnect`. The spec is claimed by the Docker-SSH lane, and that lane does not gate merges today. ## Persistence: the tombstone must survive a relaunch `closedTerminalTabTombstonesByTabId` is declared on `WorkspaceSessionState` but was missing from `workspaceSessionStateSchema` (`src/shared/workspace-session-schema.ts`), which is the load boundary for BOTH partitions -- `normalize-loaded-state-collections.ts` for `local` and `workspace-session-partitions.ts` for `ssh:<target>`. Zod strips unknown keys and the write side does not validate, so the map reached disk and was discarded on the next launch. Measured with the repo's own parser: input : closedTerminalTabTombstonesByTabId: { 'tab-1': {...} } ok = true tombstones after parse = undefined That made the fix ineffective in the exact reported scenario: close an SSH tab with the transport down, QUIT, relaunch, reconnect -- the merge runs with an empty map, the host still lists the tab, and it resurrects. "Every day I open orca and it opens more tabs" is a claim about restarts. Neither the green oracle nor the A/B control could see it: both run entirely inside one app process. It also made the 30-day TTL and the 500 cap unreachable. Fixed by adding the field with a `salvagingRecord` matching its sibling `terminalSurfaceTombstonesByPaneKey`, so one malformed entry drops that entry rather than the map. `workspace-session-schema.ts` was one line under its 300-line max-lines limit, so adding the field required room rather than a suppression (the project forbids max-lines disables and per-file bumps). Two value schemas were extracted to modules named after what they contain: `terminal-tab-id-schema.ts` and `terminal-surface-tombstone-schema.ts`. The closed-tab tombstone's own schema is colocated with its type in `closed-terminal-tab-tombstones.ts`, which is where it belongs -- omitting it from the session schema is exactly the drift that caused this bug. `workspace-session-schema-field-coverage.test.ts` is the ratchet. Two sibling tables already pin themselves with `satisfies Record<keyof WorkspaceSessionState, ...>`; this schema had no such guard and is the one that fell behind. The new file adds both halves -- a `satisfies` list that makes a forgotten field a compile error, and a runtime assertion that names it -- plus a `parseWorkspaceSession` round-trip. Without the schema entry: 3 failed. With it: 3 passed. ## A host tab the user never closed could be deleted `tabId in closedTerminalTabTombstonesByTabId` answers true for every `Object.prototype` key even on an EMPTY map, because the map is a plain object from `Object.fromEntries`. A host tab whose id is `toString` was filtered from the reconciled list, blocked from the host-unknown branch, and stripped of its layout and session id. Tab ids are validated only as non-empty and colon-free, and `createTab` honours caller-supplied id hints, so the id is reachable rather than theoretical. This was the only path in either direction that could delete a tab the user never closed. Now `Object.hasOwn`, as the same file already uses elsewhere. Suppression is also scoped structurally: `isSuppressedByClose` compares the tombstone's stored `worktreeId`, which it already carried, so it cannot reach another workspace's tab. The two sweeps that have no worktree in scope (`terminalLayoutsByTabId`, `remoteSessionIdsByTabId`) now consult the set of ids this merge actually suppressed rather than re-deriving a verdict without that scope. The scope comment at the top of the function was also wrong and is corrected. It claimed every use of suppression sits inside `replaceWorktreeIds`; it does not -- the tabs pass walks all of `orderedWorktreeIds` and the two sweeps cover the whole remote maps. What actually makes it safe is that `closeTab` strips the id from every worktree row before recording the tombstone, plus the worktree match above, plus `closeReason === 'user'` being the only writer. Real guarantee, different from the documented one. ## Divergences from open PR stablyai#16571 stablyai#16571 implements the same concept. Three deliberate changes: 1. It never retires on acknowledgement -- TTL+cap only, so it never converges. Ack retirement added. 2. It crosses the wire and lets a HOST-sourced tombstone delete a LOCAL tab in a final whole-map sweep. After stablyai#14361 that is the wrong risk; dropped. This also removes the mixed-version regression its own body flags. 3. Its hydration unions rather than replaces the map -- a union resurrects every tombstone the merge just retired, so it never converges. Its `activeTabId` nulling is also dropped as redundant: `workspace-terminal-hydration.ts:99-105,126-138` already revalidates both pointers against the tab rows it just built, and nulling twice would add a second rule that has to stay in step with the first. ## Can a tab the user did NOT close disappear? No, but the guarantee needs stating precisely. The only writer is `recordClosedTerminalTabTombstone` (`terminal-tab-close.ts:69`), reachable only on `closeReason === 'user'`; suppression additionally requires the tab not be live locally. Reopen (`recently-closed-tabs.ts:122-166`) calls `createTab` and restores cwd/shell/title/color/position, never the old id. **Caveat, stated because the slogan is not literally true:** `createTab` honours a caller-supplied id hint (`terminal-tab-creation.ts:53-65`, used by `useIpcEvents` for host-admitted tabs), so "tab ids are uuids that never recur" does not hold in this codebase. The guarantee rests on the `closeReason === 'user'` writer plus the live-local-tab check, not on id uniqueness. ## Risk Renderer-side, client-local, no wire change. The blast radius is `mergeDirectSshRemoteWorkspaceSession` and the persisted session field. Worst case if the ack logic were wrong in the retiring direction: a tombstone outlives its usefulness and suppresses a host tab whose id the host re-issues -- bounded by the live-local-tab check, the 30d TTL and the 500 cap. Worst case in the other direction is today's behaviour. `profile-project-session-field-disposition.ts` records the new field as `notRepoScoped` / `notTransferred` residue, bounded by the same TTL and cap. ## Verify pnpm test src/shared/closed-terminal-tab-tombstones.test.ts \ src/renderer/src/lib/workspace-session-closed-tab-tombstones.test.ts \ src/renderer/src/store/terminals/terminal-tab-close-tombstone.test.ts \ src/renderer/src/hooks/remote-workspace-session-merge-close-tombstones.test.ts To reproduce the bug this fixes, set `isSuppressedByClose` to `() => false` in `remote-workspace-session-merge.ts` and run `pnpm test:e2e:ssh-docker -- tests/e2e/ssh-lost-kill-tab-resurrection.spec.ts --repeat-each=2`. (cherry picked from commit 2b39165)
stablyai#16752) An enterprise user: "Every day I open orca and it opens more tabs daily at a linear scale." Three reports over a week, told on 08-19 that a PR had fixed it, reported twice more after. STA-4658 (P0), GH stablyai#12447, stablyai#15136, stablyai#10342, stablyai#9585. One install held 39 zombie tab records. The revived tab's sleeping-agent record still holds the pre-close session id, so it boots `claude --resume <old id>` -- two agents on one transcript. ## The chain, measured Reproduced deterministically in `ssh-lost-kill-tab-resurrection.spec.ts`: close an SSH tab, kill the relay daemon in the container so `pty.kill` rejects with a transport-class error, reconnect. drop 2 resurrected the closed tab <id>: baseline=1 drop1=1 drop2=2 (closed tab returned) drop3=1 The trigger is narrow and had to be measured rather than assumed: killed relay daemon reproduces **6 of 6 runs**; an orderly `ssh.disconnect` **passes**. Only an ungraceful loss -- network partition, host reboot, relay crash, a laptop sleeping mid-session -- strands the close with the RPC rejecting on a transport-class error. Both variants live in the spec behind one `runResurrectionCycles` parameterized solely by the disruption, so the difference is attributable to that single variable. What actually carries the tab back, from the pull path (`workspace.get` -> `getRemoteSnapshot`, `remote-workspace-relay-sync.ts:29`): pullSnapshot rev=3 tabs={repo:["16c4a3e1","06aba6b6"]} pullSnapshot rev=4 tabs={repo:["16c4a3e1","ff72768e"]} <- ff72768e IS the resurrected tab pullSnapshot rev=5 tabs={repo:["16c4a3e1","ff72768e","da21b76c"]} The client uploaded the session containing the tab; the user closed it; the kill RPC rejected so the close never reached the host; the host's snapshot still lists it; the client pulls it back and the merge restores it -- **correctly, by its own rule that the host is authoritative for what it knows.** A pane then mounts, respawns, and takes the recycled pty id. Client-side correlation from the same run, two controls and one positive in one run differing in exactly one variable: | Tab | Close events observed | Resurrected? | |---|---|---| | `6305cc07` | `user` + `pty-exit` | No | | `ed56f66c` | `user` + `pty-exit` | No | | `2036e760` | `user` only | **YES** | ## The fix `src/shared/closed-terminal-tab-tombstones.ts` (99 lines). A client-recorded close is first-party intent and must survive until the host acknowledges it. Per `docs/reference/ssh-execution-boundary.md` the remote verdict is `unverifiable` -- which may not authorise declaring the process dead, but equally must not authorise resurrecting the tab. This is SSH-v3 principle P2, "durable tombstones with a monotonic per-scope revision", reusing the existing `RemoteWorkspaceSnapshot.revision` rather than adding a twelfth per-tab identity field (the codebase carries eleven, 784 refs, that SSH-v3 Phase 3 deletes). - **Recorded** only on `closeReason === 'user'` (`terminal-tab-close.ts:69`). - **Suppresses** a host-sourced tab only when `tabId in tombstones && !currentTabsById.has(tabId)` -- a live local tab always wins, because deleting a live pane is the one outcome the merge exists to avoid. - **Retires** on positive acknowledgement: `!hostKnownTabIds.has(tabId) && hostRevision > observed`. Strictly newer, so a pull already in flight at close time cannot ack a close it predates. - Three never-retire guards: no revision retires nothing; a worktree the snapshot has no row for retires nothing; the first omitting snapshot only stamps the watermark. - TTL (30d) + cap (500) are **backstops** for a target the user never returns to, not the mechanism. - **Client-local only** -- never crosses the wire, so there is no mixed-version exposure. - Suppression is scoped to `replaceWorktreeIds`, which is what makes the live-tab check meaningful. A final whole-map sweep over the assembled `tabsByWorktree` would break that (a live tab is absent from `currentTabsById` outside the scope and would look suppressible); it is deliberately not there, and the comment at the top of the function says so. ## Evidence The load-bearing evidence is an A/B control on one tree, not the oracle's assertion. Flipping `isSuppressedByClose` to `false` -- one character -- reproduces the resurrection on demand: --repeat-each=2: 1) drop 2 resurrected the closed tab ab0e305d-…: baseline=1 drop1=1 drop2=2 2) drop 2 resurrected the closed tab 51533e34-…: baseline=1 drop1=1 drop2=2 2 failed With suppression on: **0 occurrences of "resurrected the closed tab" across five runs plus one independent run by a second agent.** Provenance verified positively, not by mtime: `closedTerminalTabTombstonesByTabId` appears 13x across 3 renderer chunks including `store-Do3KBvRE.js`; for every red control run `mayCreate` appeared 0 times in `out/main/index.js`. At the unit layer, disabling the same predicate: 3 failed | 39 passed. Restored: 42 passed; 287 across the workspace-session, terminal-store, remote-workspace, shared-tombstone and profile suites; 24 in the four tombstone suites. ## The oracle spec: GREEN in the full lane `ssh-lost-kill-tab-resurrection.spec.ts` passes both tests at this commit. Full Docker-SSH lane, clean tree: BUILD_SHA=49bb96e0b4c DIRTY=0 PROVENANCE tombstone=13 hasLocalTabsRow=2 hostAuthority=4 mayCreate=3 14 specs / 20 tests -> 17 passed, 2 failed, 1 skipped (10.7m) [12/20] :178 does not resurrect tabs whose kill was lost to a killed relay daemon PASSED [13/20] :190 does not resurrect tabs closed while the host is disconnected PASSED grep -c "resurrected the closed tab" (whole lane) -> 0 It passes WITHOUT PR 7 in the build (`mayCreate` present, `SshPtyAbsentFromRelayError` absent), so the bug-2 fix below is not required for it. Test 1 fails intermittently in ISOLATED single-spec runs, where a third defect blocks its cycle-2 setup. The resurrection assertion itself has never failed with this fix in place -- the intermittent failure is always a setup failure, never a resurrected tab. A reviewer running the spec alone may see it red; that is not this fix regressing. Three defects sit under STA-3374 and should not be conflated: - Bug 1 -- the closed tab resurrects. Fixed here. - Bug 2 -- `ssh-pty-session-reattach.ts:227-231` rewrites the relay's `PTY "pty-1" not found` into a bare `SSH_SESSION_EXPIRED`, so `isPtyAlreadyGoneError`'s `/PTY ".+" not found/` cannot match and `attachStablePaneOwner:242`'s already-correct fallback never runs. Owned by PR 7 (`nwparker/ssh-07-absent-from-relay`). Not required for the oracle above. - Bug 3 -- after the daemon is killed and the client launches a replacement, the client's OWN SSH transport drops and does not reconnect within 60s: no "delay step 2/9", no handshake failure, nothing. `ssh-connection.ts:1533` only logs on an SSH-level close. Unfixed, its own ticket. This is what makes test 1 intermittent in isolation. Discriminator for bug 3, measured in the isolated runs (the lane above ran without `ORCA_E2E_FORWARD_APP_LOGS=1`, so it was not re-confirmed there): `[ssh-relay] Socket probe result:` reads "DEAD" on every cycle of test 1 (daemon killed, a NEW relay must be launched) and "ALIVE" on every cycle of test 2 (daemon survived). Whenever a new daemon must be launched, the SSH transport drops afterwards and does not recover. An earlier reading blamed `kill.ts:82-84` for skipping `finishPtyShutdown` on a non-already-gone error. That was eliminated by direct test: the implied fix, `markSshRemotePtyLease(…, 'expired')` in that branch, was implemented, changed nothing, and was reverted rather than shipped unproven. Recorded so the path is not re-walked. The `SSH_SESSION_EXPIRED` rejection is real but fires during cycle 1 for the baseline pane, after which cycle 1 completes; the 60s silence begins only after `Relay channel lost ..., triggering reconnect`. The spec is claimed by the Docker-SSH lane, and that lane does not gate merges today. ## Persistence: the tombstone must survive a relaunch `closedTerminalTabTombstonesByTabId` is declared on `WorkspaceSessionState` but was missing from `workspaceSessionStateSchema` (`src/shared/workspace-session-schema.ts`), which is the load boundary for BOTH partitions -- `normalize-loaded-state-collections.ts` for `local` and `workspace-session-partitions.ts` for `ssh:<target>`. Zod strips unknown keys and the write side does not validate, so the map reached disk and was discarded on the next launch. Measured with the repo's own parser: input : closedTerminalTabTombstonesByTabId: { 'tab-1': {...} } ok = true tombstones after parse = undefined That made the fix ineffective in the exact reported scenario: close an SSH tab with the transport down, QUIT, relaunch, reconnect -- the merge runs with an empty map, the host still lists the tab, and it resurrects. "Every day I open orca and it opens more tabs" is a claim about restarts. Neither the green oracle nor the A/B control could see it: both run entirely inside one app process. It also made the 30-day TTL and the 500 cap unreachable. Fixed by adding the field with a `salvagingRecord` matching its sibling `terminalSurfaceTombstonesByPaneKey`, so one malformed entry drops that entry rather than the map. `workspace-session-schema.ts` was one line under its 300-line max-lines limit, so adding the field required room rather than a suppression (the project forbids max-lines disables and per-file bumps). Two value schemas were extracted to modules named after what they contain: `terminal-tab-id-schema.ts` and `terminal-surface-tombstone-schema.ts`. The closed-tab tombstone's own schema is colocated with its type in `closed-terminal-tab-tombstones.ts`, which is where it belongs -- omitting it from the session schema is exactly the drift that caused this bug. `workspace-session-schema-field-coverage.test.ts` is the ratchet. Two sibling tables already pin themselves with `satisfies Record<keyof WorkspaceSessionState, ...>`; this schema had no such guard and is the one that fell behind. The new file adds both halves -- a `satisfies` list that makes a forgotten field a compile error, and a runtime assertion that names it -- plus a `parseWorkspaceSession` round-trip. Without the schema entry: 3 failed. With it: 3 passed. ## A host tab the user never closed could be deleted `tabId in closedTerminalTabTombstonesByTabId` answers true for every `Object.prototype` key even on an EMPTY map, because the map is a plain object from `Object.fromEntries`. A host tab whose id is `toString` was filtered from the reconciled list, blocked from the host-unknown branch, and stripped of its layout and session id. Tab ids are validated only as non-empty and colon-free, and `createTab` honours caller-supplied id hints, so the id is reachable rather than theoretical. This was the only path in either direction that could delete a tab the user never closed. Now `Object.hasOwn`, as the same file already uses elsewhere. Suppression is also scoped structurally: `isSuppressedByClose` compares the tombstone's stored `worktreeId`, which it already carried, so it cannot reach another workspace's tab. The two sweeps that have no worktree in scope (`terminalLayoutsByTabId`, `remoteSessionIdsByTabId`) now consult the set of ids this merge actually suppressed rather than re-deriving a verdict without that scope. The scope comment at the top of the function was also wrong and is corrected. It claimed every use of suppression sits inside `replaceWorktreeIds`; it does not -- the tabs pass walks all of `orderedWorktreeIds` and the two sweeps cover the whole remote maps. What actually makes it safe is that `closeTab` strips the id from every worktree row before recording the tombstone, plus the worktree match above, plus `closeReason === 'user'` being the only writer. Real guarantee, different from the documented one. ## Divergences from open PR stablyai#16571 stablyai#16571 implements the same concept. Three deliberate changes: 1. It never retires on acknowledgement -- TTL+cap only, so it never converges. Ack retirement added. 2. It crosses the wire and lets a HOST-sourced tombstone delete a LOCAL tab in a final whole-map sweep. After stablyai#14361 that is the wrong risk; dropped. This also removes the mixed-version regression its own body flags. 3. Its hydration unions rather than replaces the map -- a union resurrects every tombstone the merge just retired, so it never converges. Its `activeTabId` nulling is also dropped as redundant: `workspace-terminal-hydration.ts:99-105,126-138` already revalidates both pointers against the tab rows it just built, and nulling twice would add a second rule that has to stay in step with the first. ## Can a tab the user did NOT close disappear? No, but the guarantee needs stating precisely. The only writer is `recordClosedTerminalTabTombstone` (`terminal-tab-close.ts:69`), reachable only on `closeReason === 'user'`; suppression additionally requires the tab not be live locally. Reopen (`recently-closed-tabs.ts:122-166`) calls `createTab` and restores cwd/shell/title/color/position, never the old id. **Caveat, stated because the slogan is not literally true:** `createTab` honours a caller-supplied id hint (`terminal-tab-creation.ts:53-65`, used by `useIpcEvents` for host-admitted tabs), so "tab ids are uuids that never recur" does not hold in this codebase. The guarantee rests on the `closeReason === 'user'` writer plus the live-local-tab check, not on id uniqueness. ## Risk Renderer-side, client-local, no wire change. The blast radius is `mergeDirectSshRemoteWorkspaceSession` and the persisted session field. Worst case if the ack logic were wrong in the retiring direction: a tombstone outlives its usefulness and suppresses a host tab whose id the host re-issues -- bounded by the live-local-tab check, the 30d TTL and the 500 cap. Worst case in the other direction is today's behaviour. `profile-project-session-field-disposition.ts` records the new field as `notRepoScoped` / `notTransferred` residue, bounded by the same TTL and cap. ## Verify pnpm test src/shared/closed-terminal-tab-tombstones.test.ts \ src/renderer/src/lib/workspace-session-closed-tab-tombstones.test.ts \ src/renderer/src/store/terminals/terminal-tab-close-tombstone.test.ts \ src/renderer/src/hooks/remote-workspace-session-merge-close-tombstones.test.ts To reproduce the bug this fixes, set `isSuppressedByClose` to `() => false` in `remote-workspace-session-merge.ts` and run `pnpm test:e2e:ssh-docker -- tests/e2e/ssh-lost-kill-tab-resurrection.spec.ts --repeat-each=2`.
ELI5
Closing a terminal tab on an SSH workspace leaves no record that it was closed — the host snapshot just stops (or never starts) listing it. Since "absent" can also mean "not uploaded yet", the sync merge plays it safe and keeps re-inserting tabs the host still lists, so closed tabs come back after every sleep/wake reconnect. This PR makes closing a tab leave a small tombstone ("this tab id was closed at T"), carried in the session and the host snapshot, so the merge can finally tell "closed" apart from "never uploaded" — closed tabs stay closed, and no live tab is ever deleted by mistake.
What Changed
closeTabrecords a tombstone{closedAt, worktreeId}keyed by tab id for deliberate closes (user/cleanup; a pty-exit close is the process ending, not the user retiring the tab).closedTabTombstonesByTabId(worktreePath-scoped on export, resolved back on import), and the client's inbound snapshot normalizer preserves the field. The relay stores the session opaquely (verified:workspace-session-handler.tsreplace-session does no field whitelisting).mergeDirectSshRemoteWorkspaceSessionunions both sides' tombstones (newerclosedAtwins), never re-inserts a tombstoned remote tab (reconcile, host-unknown re-add, layouts, remoteSessionIds, final sweep), and a live local tab invalidates a stale tombstone for its id each pass. Fixtures without tombstones behave exactly as before — the local-survival suite passes unchanged, and a new test pins that trade.Why
Live reproduction (Linux SSH relay host): the client UI showed 5 tabs for a worktree while the host's
~/.orca/sessions/<ns>.jsonheld 14 — ghosts accumulated over days. The client kept re-pushing the 14-entrytabsByWorktreePathon every revision bump (live tabs' title spinner frames updated in the snapshot in real time) while closes and renames never landed. On reconnect the pull replaced the legacy tab list wholesale, andreconcileWorktreeTabModelre-promoted the ghosts into the visible tab strip on the next tab operation — a self-reinforcing loop.The root cause is that the snapshot has no way to express "closed": the merge's documented trade (absence ≠ closed,
remote-workspace-session-merge.ts) is correct for protecting not-yet-uploaded tabs, but it needs an explicit close signal to stop resurrecting deliberately closed ones. Tombstones add that signal without touching the trade — which is also the direction proposed in #12447 ("record the intent … never silently drop it").Relationship to existing PRs on #10342: #10365 marks retired pty ids locally, which sweeps dead-session tabs on one client but the marker never reaches the host, so other clients and post-session-loss restores still resurrect; #12339 skips pulls whose revision hasn't advanced, which prevents fresh poisoning but cannot converge an already-poisoned session that keeps advancing its own revision (the state observed above). Tombstones travel with the snapshot, so both the propagation and the already-poisoned cases converge; the write-drop fix here matches #12339's defer approach.
Mixed versions: an old client pulling a new snapshot ignores tombstones and re-pushes a session without the field (replace-session semantics), re-arming the loop until clients update — inherent to any additive field, called out so it's a known trade. The pre-existing sibling gap (
defaultTerminalTabsAppliedByWorktreePathis also dropped by the inbound normalizer) was left untouched as out of scope.Linked Issue
Fixes #10342
Visual Proof
BEFORE (current release, macOS client + Linux SSH host) — recording starts right after reconnecting from a disconnect, with 5 tabs open. At 0:07 one tab is closed (its
×is clicked); instead of just that tab disappearing, moments later the tab strip floods with previously closed tabs resurrecting (their original persisted titles):before-trimmed.mp4
AFTER: with this branch the same flow leaves closed tabs closed; verified host-side (the snapshot's
tabsByWorktreePathconverges to the visible tabs and gainsclosedTabTombstonesByTabId). An AFTER recording from a fix build can be added on request before/while marking ready for review.Testing
Automated (all run on the Linux SSH-host side of the repro):
closeTabrecording (user vs pty-exit), session patch/payload round-trip, projection export/import (target scoping, unresolvable drop), merge behavior (ghost not re-inserted; remote tombstone removes a session-persisted entry; live tab beats stale tombstone;preserveLocalTerminalTabIds∩ tombstone precedence; absence≠closed unchanged), inbound normalization survival + tombstone-only delta still pushes, suppressed-write defer.pnpm vitestoversrc/shared,src/renderer/src/hooks,src/renderer/src/lib,src/renderer/src/store: 13,556 passed / 62 skipped / 0 failed;oxlint,typecheck:web,typecheck:nodeclean.pnpm test: 58,510 passed; the 13 failures (src/main/browser/*cookie*.electron.test.ts,src/main/ssh/ssh-connection-host-key-store-wiring.test.ts) reproduce identically with this branch's sources reverted to basedeae7212d9— pre-existing on main in this environment, unrelated to this change.Manual: reviewer repro — on an SSH workspace, open tabs, sleep/disconnect, close tabs, reconnect, then open/close a tab: closed tabs stay closed; the host snapshot gains
closedTabTombstonesByTabIdand itstabsByWorktreePathconverges to the visible tabs.AI Disclosure
Developed with Claude Code (Claude Fable 5). Root cause was established by live reproduction against a real SSH host before any code was written; every change was implemented TDD (failing test first), independently re-reviewed, and the final branch review's findings were fixed and re-verified.
Review
Agent skill upstream boundary
docs/reference/agent-skill-sharing-upstream-boundary.mdand copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.Notes
workspace-session.tshelper extractions exist only to stay under the oxlintmax-linescap; no logic changed.Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)